Crowdsourcing is the technique in which a computational process performs its function by outsourcing certain steps to one or more entities. These entities may include humans, computers, or combinations thereof. For example, a computational process may use crowdsourcing by asking entities to provide an answer to a question. The answer to the question may then be used by the computational process in performing a task. Platforms that use crowdsourcing have become a crucial infrastructure for distributing, collecting, and aggregating human judgment for a wide range of tasks and applications.
Current systems that utilize crowdsourcing typically have limitations that fix in advance the amount of resources (i.e., entities) that may be used for each task. If too many entities are queried, then the cost becomes too high and if too few are queried then the result may not be accurate enough. Obtaining a tradeoff between cost and accuracy may be difficult, especially if the set of tasks are heterogeneous.